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Abstract 

NASA is investigating revolutionary crew-vehicle interface technologies that strive to 
proactively overcome aircraft safety barriers that would otherwise constrain the full realization of 
the next-generation air transportation system. A fixed-based piloted simulation experiment was 
conducted to evaluate the complementary use of Synthetic and Enhanced Vision technologies. 
Specific focus was placed on new techniques for integration and/or fusion of Enhanced and 
Synthetic Vision and its impact within a two-crew flight deck on the crew’s decision-making 
process during low-visibility approach and landing operations. Overall, the experimental data 
showed that significant improvements in situation awareness, without concomitant increases in 
workload and display clutter, eould be provided by the integration and/or fusion of synthetic and 
enhanced vision technologies for the pilot-flying and the pilot-not-flying. During non-normal 
operations, the ability of the crew to handle substantial navigational errors and runway incursions 
were neither improved nor adversely impacted by the display concepts. The addition of 
Enhanced Vision may not, unto itself, provide an improvement in runway incursion detection 
without being specifically tailored for this application. Existing enhanced vision system 
procedures were effectively used in the crew decision-making process during approach and 


missed approach operations but having to forcibly transition from an excellent FLIR image to 
natural vision by 100 ft above field level was awkward for the pilot-flying. 
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Commercial Flight Crew Decision-Making during Low- Visibility Approach Operations using 

Fused Synthetic / Enhanced Vision Systems 
The United States air transportation system is undergoing a transformation to 
accommodate a projected 3-fold increase in air operations by 2025 (Joint Planning and 
Development Office, 2004). Technological and systemic changes are being developed to 
significantly increase the capacity, safety, efficiency, and security for this Next Generation Air 
Transportation System (NextGen). One of the key capabilities envisioned to achieve these goals 
is the concept of Equivalent Visual Operations (EVO), whereby Visual Flight Rules (VFR) 
operational tempos and procedures (e.g., separation assurance) are maintained independent of the 
actual weather conditions. One approach to attain the goal of EVO would be the creation of a 
virtual visual flight environment for the flight crew, independent of the actual outside weather 
and visibility conditions, through the application of Enhanced Vision (EV) and Synthetic Vision 

(SV) technologies. 

NASA is investigating revolutionary crew/vehicle interface (CV1) technologies that have 
the potential to optimize situation awareness and reduce the propensity for, and minimize the 
risks associated with, pilot error while proactively overcoming aircraft safety barriers that would 
otherwise constrain the full realization of the NextGen (National Aeronautics and Space 
Administration, n.d.). Part of this research effort involves the use of EV and SV systems and 
other interface modalities as enabling technologies to meet the safety challenges of the NextGen 
EVO operational concept - that is, having the safety and capacity rates of present-day VFR 
operations in Instrument Meteorological Conditions (1MC). 
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Synthetic and Enhanced Vision 

SV is a computer-generated image of the external scene topography, generated from 
aircraft attitude, high-precision navigation, and data of the terrain, obstacles, cultural features, 
and other required flight information. SV provides significant improvements in terrain 
awareness and reductions in the potential for Controlled-Flight-Into-' Terrain incidents/accidents 
compared to current cockpit technologies (Kramer, Arthur, Bailey, & Prinzel, 2005; Arthur, 
Prinzel, Kramer, Bailey, & Parrish, 2003; Schiefele el al., 2005; Schncll, Thcunissen, & 
Rademaker, 2005). 

EV (or Enhanced Flight Vision System, EFVS) is an electronic means to provide an 
image of the forward external scene topography by use of an imaging sensor, such as a Forward- 
Looking InfraRcd (FLIR) or millimeter wave radar (MMWR). EV during low-visibility 
approach conditions provides significant improvements in flight performance, pilot workload 
reduction, and decreases in the propensity for missed approaches (e.g., see Connor & Mages, 
1993). 

The intended use of EV and SV technology mirror each other as they both attempt to 
eliminate low-visibility conditions as a causal factor to civil aircraft accidents and replicate the 
operational benefits of clear-day flight operations, regardless of the actual outside visibility 
condition. The methodology by which this capability is achieved through SV or EV, however, is 
significantly different. While some may consider the technologies to be competing; they are, in 
fact, complementary (Arthur, Kramer, & Bailey, 2005). 

SV, by virtue of being weather-independent and unlimited in field-of-rcgard, is 
particularly advantageous during flight phases, such as approach, which may be obscured by 
clouds and precipitation of which an EV sensor cannot penetrate. Recognition of terrain and 
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cultural features may also be improved over an EV view since die display presentation is 
optimized by the display designer, not the product of the sensor and its environment. Pilot 
recognition of EV terrain and cultural features depends upon the reflected, emitted, and / or 
refracted energy at the spectral frequencies of the EV sensor and the ability of the pilot to 
(correctly) interpret this image. Atmospheric effects, time of day, and sensor characteristics can 
be important factors in the quality of the EV imagery. On the other hand. EV provides a direct 
view of the vehicle external environment; independent of the derived aircraft navigation solution 
or of a database. Under conditions of smoke, haze, and night, a FL1RZEV provides orders-of- 
magnitude improvement over the pilot's natural vision; greatly enhancing the pilot's situation 
awareness and reducing the pilot's workload. The comparison of SV and EV, as shown in 
Figure 1, on a night visual meteorological conditions (VMC) approach into an airfield highlights 
the similarities and differences in these two technologies. 

Synthetic Vision 


Figure I. Synthetic vision and enhanced vision comparison. 

Past Research 

Previous synthetic vision research (Parrish, Busqucts, Williams. & Nold, 2003) has 
shown that a "flight-critical" synthetic vision implementation which uses automated decision 
aiding functions for object detection and database alignmenl/navigation error detection produces 
superior performance to synthetic vision concepts with an EV inset display. To date, however. 
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technology for "perfect” object detection and database/navigation error detection does no£ exist. 
Further, even if these systems come to fruition, there may still be gaps, such as minimal radar 
cross section objects or bclow-threshold detection values, which may require other additional 
integrity and error cheeks. 

SV with EV inset displays may offer one possible method to provide the pilot with 
information sufficient to perform navigation integrity and obstacle clearance checks. While 
these concepts are viable, performance and pilot workload (Parrish et al., 2003) suffer in 
comparison to automated methods to achieve these same capabilities. Other studies have shown 
similar results (McKay, Guirguis, Zhang, & Newman, 2002). Object detection by pilots was 
found to be best using a dedicated EV display, as opposed to a "shared” display, particularly one 
that did not include symbology. (However, the presence or absence of symbology was not 
tested.) From this study and others, the ability of pilots to perform navigation integrity checks 
and obstacle identification principally depends upon the visual acuity provided by the displayed 
imagery for the pilot (BolT & Lincoln, 1988), as affected by the resolution and acuity of the 
sensor and display systems; the characteristics of the object and its surrounding scene or 
background features; the display clutter; display size; and the display and object color and 
contrast characteristics. 

While EV might improve SV operations, the converse warrants investigation as well. In 
2004, Section 91.175 of the Federal Aviation Regulations was amended such that operators 
conducting straight-in instrument approach procedures (other than Category II or Category III) 
may now operate below the published Decision Height (DH) and Minimum Descent Altitude 
(MDA) when using an approved EFVS on the pilot's Head-Up Display (HUD). This rule change 
now provides "operational credit” for EV. As such, EV operations will become more prevalent. 
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No such credit currently exists for SV. However, SV may be advantageous during flight phases, 
such as approach, in aiding the pilot's awareness of terrain, obstacles, and flight path which may 
be obscured by clouds and precipitation of which an EV sensor cannot penetrate. SV may also 
provide the crew with “visual momentum” to assist the crew's understanding and correct 
interpretation of the EV sensor imagery. 

Cur rent Study 

A Fixed-based simulator experiment was conducted to evaluate the complementary use of 
SV and EV technologies, specifically focusing on new techniques for integration and/or fusion of 
synthetic and enhanced vision technologies and its influence on crew coordination during low- 
visibility approach and landing. The objective of this experiment was to test the utility, 
acceptability, and usability of integrated 1 fused EV and SV technology concepts in a two-crew 
commercial or business aircraft cockpit; these results are described in Bailey, Kramer, and 
Prinzcl (2006). The current paper describes experimental results specific to crew decision- 
making during low- visibility approach and crew response to non-normal events that were staged 
in this experiment using a fused synthetic/enhanced vision system. 

Method 

Subjects 

Twenty-four pilots, representing seven airlines and a major cargo carrier, participated in 
the experiment. All subjects had previous experience flying with HUDs. The subjects had an 
average of 1787 hours of HUD flying experience and an average of 13.8 years and 16.2 years of 
commercial and military flying experience, respectively. EV experience was not required 
although some pilots were familiar with imaging sensor technology from prior military flight 
experience. None of the subjects were flying EV in current operations. 
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Simula tor 

The experiment was conducted in the Integration Flight Deck (IFD) simulation facility at 
NASA Langley Research Center (LaRC). The IFD emulates a Boeing B-757-200 aircraft and 
provides a full-mission simulator capability. The collimated out-the-window (OTW) scene is 
produced by an Evans and Sutherland ESIG 4530 graphics system providing approximately 200 
degrees horizontal by 40 degrees vertical field-of-vicw (FOV) at 26 pixels per degree resolution. 
The OTW imagery used the same source data as the SV database. The subjects occupied the left 
(as pilot-living, PF) and right (as pilot-not-flying, PNF) seats. The left seat included an overhead 
HUD projection unit and the right seat included an auxiliary display (AD) under the right side 
window. Traditional primary flight and navigation displays were presented head-down. 

Head- Up Display 

The HUD subtended approximately 32 c horizontal by 24° vertical FOV. The HUD 
presentation was written strictly in a raster format from a video source (RS-343) input. The 
input consisted of a video mix of symbology and computer-generated scene imagery (either EV 
or SV as described in the following). The symbology included "haloing” to ensure high-contrast 
symbology against the scene imagery background. Brightness and contrast controls were 
provided to the pilot. Also, the pilot had a declutter control, implemented as a push-button on 
the left hand horn of the PF yoke. The button cycled through three "declutter” states: 1 ) No 
declutter (full symbology and scene imagery); 2) "Raster" declutter (full symbology, no scene 
imagery); and 3) "Full declutter” (no HUD display). 

Four HUD display concepts (see Figure 2) were tested, differing from each other in: 1 ) 
the type of raster background presented; and, 2) in the type of symbology presented. Two raster 
formats were flown, either EV only (hereinafter referred to as "FL1R”) or a fusion SV/EV image 
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(hereinafter referred to as "Fusion.") The FLIR HUD concept represented our "baseline" EFVS 
HUD condition. In this configuration, the simulated FLIR output was exclusively displayed 
whether useful imagery of the external scene topography was being provided or not. The Fusion 
HUD concept represents one method of providing complementary SV/EV information for the 
pilot flying. The Fusion raster started out as unadulterated SV imagery, transitioning through a 
fused SV/EV presentation beginning at 600 ft ( 183 m) above field level (AFL), and ending with 
unadulterated FLIR imagery by 500 ft (152 m) AFL. Between 600 feet and 500 feet AFL, a step 
function modulated the fusion from 100% SV/0% EV ending at 0% SV/100% EV. 

Each raster concept showed FLIR-only below 500 ft to take advantage of the operational 
credit now offered by use of FLIR on the HUD. The 500 0 transition altitude was chosen from a 
usability study prior to the test and flight experience (Kramer, Arthur, et al., 2005) balancing the 
following factors: 1 ) FLIR imagery is required no lower than 200 ft (61 m) AFL to provide an 
operational credit for FLIR usage, yet there is sufficient time (from 500 ft to 200 ft) to become 
acclimated to the FLIR imagery, 2) SV imagery for the Fusion concept can assist the PF in 
establishing and maintaining stabilized approach parameters (the minimum recommended IMC 
stabilized approach altitude is 1000 ft (305 m) AFL); and. 3) 500 ft AFL is at or just after the 
recommended minimum VMC stabilized approach altitude to allow full utilization of EV. The 
100 ft (30.5 m) transition between SV and EV for the Fusion concept imparts visual momentum 
between the different image sources for the PF HUD. 

The two raster formats (FLIR. Fusion) were factorially combined with two symbology 
sets: ( 1 ) The standard HUD symbology (hereafter referred to as "Baseline") and (2) the same 
standard HUD symbology enhanced with pathway guidance and a runw'ay outline (hereafter 
referred to as the ‘Tunnel” symbology set). The "tunnel” symbology set (Kramer, Prinzcl, 
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Arthur, & Bailey, 2005) was tailored to transition at the same altitude of 500 ft AFL as the 
Fusion raster. The tunnel ended at 500 ft AFL to minimize clutter and was replaced by the 
glideslope reference line to ensure approach path angle awareness. Below 500 ft, the only 
additional symbology in the Tunnel symbology set, over and above that of the Baseline set, was 
the runway outline symbol. (Note that below 500 ft, the FLIR-Baseline and Fusion-Baseline 
configurations were identical.) 


FLIR Raster 


Fusion (SVS) Raster 
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Figure 2. Head-Up Display (HUD) formats on approach at 7(H) feet AGL 
In Figure 2, all four PF-HUD concepts are shown - the FLIR-Baseline, FLIR-Tunnel, 
Fusion-Baseline, and the Fusion- Tunnel. In the FLIR-Baseline concept, a minimum of 
symbology is used and the FLIR does not necessarily provide terrain and runway cues. (The 
FLIR imagery depends, for instance, upon the simulated atmospheric conditions.) Conversely, 


the Fusion-Tunnel concept uses tunnel guidance for distinct path demarcation and SV for clear 
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terrain and runway references, above 500 A AFL. Below 500 ft with the Tunnel symbol set, the 
runway outline provides an element of SV, creating "visual momentum" within the EV raster 
background image and also, a direct indication of the navigation solution accuracy if the runway 
is visible in the FL1R imagery. 

Auxiliary Display 

The PNF-Auxiliary Display (PNF-AD) was located outboard of the PNF location. The 
display was positioned as a compromise between optimal PNF viewing position, minimal 
display/instrument panel obscuration, and moderate installation complexity. The 8.4 in (21 cm) 
diagonal display was full-color w ith 1024 x 768 pixel resolution. The display video source was a 
video mix of "haloed" symbology and computer-generated scene imagery (either EV or the 
output of a fused EV/SV signal as described below). 



Figure 3. Four Auxiliary Display (AD) formats on final. 
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Four PNF-AD display concepts (see Figure 3) were tested, differing from each other in: 

1 ) the type of raster background presented; and. 2) the type of symbology presented. The raster 
was either EV only (hereinafter referred to as "FLIR”) or a fused SV/EV image (hereinafter 
referred to as “Fused”). The symbology was either "On" or “OfT for the data runs. When 
present, the symbology was a subset of the standard HUD symbology (see Figures 3). 

The AD- fused raster image was pilot-controllable and could be controlled throughout the 
approach to one of 10 states: FLIR only. SV only, or 8 fusion combinations of FLIR and SV, 
using an Equinox EP-3000™ fusion board. The fusion employs a feature-level extraction 
algorithm with two pilot control inputs: ( 1 ) feature-level fusion of FLIR and SV and (2) 
modulation of false-color coding of the fusion image. 

In Figure 3. examples of the four PNF-AD concepts are shown: FLIR-No Symbology 
(upper left). FLIR-Symbology (lower left), Fused-No Symbology (upper right), and Fused- 
Symbology (lower right). 

Evaluation Task 

The evaluation task was selected to approximate what may be typical of the emerging 
NextGen concept called an “equivalent visual operation.” The task was based on a published 
visual arrival - reflecting an efficient and preferred routing for Air Traffic Control ( ATC) and 
noise-abatement - which currently requires VMC for the pilot to see-and-avoid terrain, traffic, 
and obstacles while navigating with respect to ground references. The approach path is not too 
dissimilar from a Required Navigation Performance (RNP)-type arrival, requiring a curved, 
descending path. The evaluation task tests the ability of SV and EV technologies to support this 
type of operation by providing “equivalent visual” information into the cockpit. Further, if this 
technology succeeds in providing a visual arrival capability, the potential for operational 
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efficiency and minimums reduction above and beyond what can be provided by RNP may be 
offered. 

The PF hand-flew the base and final leg portions of the Sparks Visual Arrival to RNO 
(FA A identifier for Reno/Tahoe International Airport) Runway 16R (16 Right) with autothrottles 
engaged at an approach speed of 138 knots. The Sparks Visual Arrival requires VFR conditions 
under today's operating rules. However, to test the efficacy of the concepts for EVO capability, 
visibility was reduced significantly to 1MC. The aircraft was configured for landing prior to each 
run (landing gear down and Haps 30 degrees). The path converged into the Instrument Landing 
System (1LS) for Runway 16R. The aircraft was configured to land and, if properly flown, the 
aircraft would be on the 1LS and on a stabilized approach by 1000 ft (305 m) AFL. The PNF 
monitored the approach from the right-hand side of the flight deck using standard instruments 
and the AD. Subjects were instructed that the run would end at main gear touchdown but that 
they should perform a go-around if they felt the landing was not safe. 

Enhanced Vision System Crew Procedures 

To aid crew decision-making during low-visibility approach operations, enhanced vision 
system (EVS) crew procedures (adapted from those currently used in business aircraft EVS 
operations) were established. Instructions in the use of the procedures were given to each crew. 
An overview of these procedures is given in Table 1, including automatic call-outs. The altitude 
call-outs were set-up assuming a 200 ft DH for the published, non-EVS approach. (A "flat- 
earth" model was simulated so differences between the barometric altitudes and radar altitudes 


for decision altitudes’heights were inconsequential.) 
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Table 1 

E VS Crew Procedures 


Altitude- 

Based 

Events 

AFL' 
Barometric- 
Altitudes (ft) 

Automatic 

Callouts 

Pilot Flying (PF) Tasks 

Pilot Not Flying (PNF) Tasks 

500 feet 
AFL 

500/4912 

“5 00" 

Response: "Systems Normal. 
EVS Normal ~ 

Call "500 Jeer 

lOOfcet 
Above 
M miniums 

300/4712 

" Approaching 
Minimums" 

Response: "Check" 

Call “110 feet Above" 

Published 
Minimums 
(200 ft AFL) 

200/4612 


With EVS Visual Cues. 
Call "EVS Ughts" 

Without EVS Visual Cues. 
Call "doing Around" 

When Visual Cues Appear 
Call "Lights" or "Field in SighC 

Without PF Call of * EVS Lights’. 
Call "do Around" 

EFVS 

Decision 

Altitude 

( 100 ft AFL) 

100/4512 


When Actual Visual Cues. 
Call “ Landing ” 

Without Actual Visual Cues, 
Call "doing Around " 

When Visual Cues Appear. 
Call "Lights" or "Field in Sight" 

Without PF Call of ‘Landing'. 
Call "do Around" 


At 500 ft AFL, the “EVS Normal, System Normal” call by the PF corresponds to the 
point where the PF would nominally cheek that the EVS was set-up properly and functioning 
properly on the approach. In our test, this call-out cued the PF to ensure that the HUD declutter, 
brightness and contrast were properly set. No EVS controls were available to the llight crew in 
the experiment. The EPs were instructed that the FLIR was always set to its optimal sensor 
setting for the approach and landing. 

By the established decision height of 200 ft (AFL), the crew procedures dictate that the 
PF must have the required EFVS references or the required landing visual references (using 
natural vision) to continue the descent. The landing references were those published in FAR 
§91.175. For this test, the approach light system for RNO 16R provided the prominent EFVS 
references. If these EFVS references were visible, the PF was instructed to call "EVS Lights". 
If the PF saw the lights or markings of the threshold (the predominant landing visual reference 
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for RNO 16R), the FF called “Landing.” The “landing" call was required no later than 100 ft 
AFL. 

The PNF provided monitoring, including back-up on all decision heights, and was 
instructed to call "go-around" if "EVS Lights” was not called at or before 200 ft DH or if 
"Landing” was not called by 100 ft DH. The PNF was allowed to assist the FF in picking up the 
required visual cues (normal or EFVS). Transfer of control between the Captain and First- 
Officer was not permitted. 

The crew procedures were new to all of the flight crews. Some procedures were counter 
to, others consistent with, their current airline Standard Operating Procedures (SOPs). In either 
case, the crew procedures were trained and "enforced” for the test as specific guidance on crew 
tasks and expected crew responses to assist decision-making for EVS-approach and landing 
operations. During the post-test debrief, questions and issues of how these procedures may or 
may not work within their airline operation and SOPs were discussed. Flight crews from the 
same airline were paired to the greatest extent possible to minimize SOP differences and to 
mitigate potential interference in crew interaction. 

Experiment Matrix 

Nominally, forty experimental runs were completed by the evaluation crew with each 
pilot flying 20 approaches evaluating the HUD concepts and with each pilot monitoring 20 
approaches while evaluating the AD concepts. 

The wind and weather varied on each run. The nominal visibility in the LVS and OTW 
varied from 1 mile down to 0.5 mile ( 1 .6 to 0.8 km). The required EVS visual references 
became visible on the HUD between 450 ft ( 137 m) and 250 ft (76 m) AFL. Four runs per flight 
crew were specifically designed so the EVS visual references were visible but the required 
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runway (normal vision landing) references were not. With good crew decision-making and in 
accordance with the proper use of the EVS crew procedures, these four runs should conclude by 
a go-around initiated no lower than 100 ft AFL. 

The PF was instructed to fly each approach as precisely as possible using the display 
information available, as the effect of the display information on the PF’s ability to fly the 
approaches would be quantitatively and qualitatively evaluated. In addition, the PF was 
instructed to land as close as possible to the centerline of the runway. 

Rare Even t Testing 

A significant component of the test, in addition to the nominal runs, was met by 
measuring the ability of the flight crew to react and properly handle non-normal events. Four 
non-normal runs were flown by each crew. The non-normals were runway incursion <RI) 
scenarios and database integrity monitoring scenarios. The number and ordering of R1 and 
database integrity scenarios were designed to avoid expectancy on the part of the flight crew 
(Foyle & Hooey, 2003). 

The Rl scenarios simulated an incursion with either a non-transponding baggage cart or 
fire truck. Both vehicles were stationary and positioned in the same location, approximately 850 
ft (260 m) from the RNO Runway 16R landing threshold and just slightly offset from the 
centerline. They were both positioned perpendieular to the runway (i.c., they were facing toward 
the runway edgelines.) The weather on the runway incursions was held constant at 2400 ft (732 
m) runway visual range (RVR) OTW with the lowest cloud layer at 500 ft (152 m) AFL. The 
FL1R visibility was very good in this condition - approximately 4 times the OTW RVR. The 
baggage cart runway incursion was always performed before the fire truck incursion. The 
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baggage cart was much more difficult to see due to its small size. This ordering tested for "just 
noticeable differences" for runway incursion detection. 

The database integrity monitoring scenarios purposefully introduced a lateral navigation 
solution error (of either 50 or 75 feet { 15 or 23 m{) with respect to the real runway. This error 
resulted in the synthetic vision terrain, pathway and guidance cue being misaligned from the 
FL1R and ILS data (which were defined in the flight crew briefing as being correct). 

Procedure 

The subjects were given a 1-hour briefing to explain the SV/EV concepts on the HUD 
and AD, EVS crew procedures, and the expected evaluation tasks. After the briefing, a 2-hour 
training session in the 1FD was conducted to familiarize the subjects with the aircraft handling 
qualities, display symbologies, EVS crew procedures, and controls. The Tare-event’ scenarios 
were not discussed, although the pilot's responsibility for maintaining safe operations at all times 
was stressed. Data collection lasted approximately 4.5 hours followed by a 30-minute semi- 
structured interview. The pilots were also given a take-home final questionnaire. The entire 
session including lunch and breaks lasted approximately 9 hours. 

Results 

The focus of this journal article is on the crew decision-making process on approach 
while evaluating integrated/ fused enhanced and synthetic vision technologies and on the crew's 
reaction to non-normal flight situations (navigation errors and runway incursions). A thorough 
discussion of the pilots' path control performance (lateral and vertical) and pilot ratings of 
mental workload, situation awareness, display clutter, and display preferences are detailed in 
Bailey, et al. (2006). Overall, the experimental data showed that significant improvements in 
pilot situation awareness (SA), without concomitant increases in workload and display clutter. 
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could be provided by the integration and/or fusion of synthetic and enhanced vision technologies 
for the PF and the PNF. Similarly, SA improvements without workload or display clutter 
penalties were found by the addition of the tunnel with the HUD imagery for the PF and by the 
addition of fusion control and symbology on the AD for the PNF. 

Illegal Landings and Their Effect on Crew Decision-Making 

To test crew-decision making on approach, each flight crew was confronted with four 
trials where weather conditions obscured the required visual cues necessary to complete the 
landing as defined by FAR §91.175. Of the 48 “illegal landing" trials, during only six of these 
trials did pilots continue and land the aircraft. These data are tabulated in Table 2. identifying 
the crew, the display concept being flown, and what the flight crew were observed doing, as 
opposed to initiating a go-around. 

Table 2 


Illegal Landing Trials 


Crew 

Observation 

PF-HUD Concept 

PNF- AD Concept 

3 

Go-around considered 

FLlR. No Tunnel 

Fused/No Symbology 

4 

Threshold lights called at 80 fi AFL 

1 V 

FLIR/No Tunnel 

FLlR'Syinbology 

4 

Threshold lights called at 60 ft AFL 

FLIR'No Tunnel 

Fused’’ Symbology 

7 

Threshold lights called at 100 ft AFL 

FLIR/Tunnel 

Fused’’ Symbology 

8 

Threshold lights called at 120 ft AFL 

FLIR'No Tunnel 

1 

Fused/No Symbology 

ii | 

Threshold lights called at 90 ft AFL • 
: 

FLIR/No Tunnel • 

FLIRNo Symbology 


The results demonstrated a small but definite potential (12.5%) for flight crews to 
continue approaches to a landing during visibility conditions that instead require a go-around 
under the §91.175 operational rules for EFVS. On all of the illegal landing scenarios, the pilot 
Hying had excellent visibility of the runway using the FLIP on the HUD. but wouldn't have been 
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able to see the runway visual landing references until descending below 100 ft AFL. When 
"threshold lights" were called by the EPs above 80 ft AFL, this observation was either erroneous 
(i.e., they could not in fact see the threshold lights using natural vision) or they saw the threshold 
lights in the EFVS image itself and were confused about which was EFVS and which was natural 
vision. (The current §91.1 75 EFVS rule requires visual acquisition of the runway references 
without use of the EFVS by 100 ft height above touchdown, or HAT.) 

No statistically significant effects of symbology type on the HUD concept was observed, 
although 5 of the 6 events occurred when the runway outline was not present. 

The operational procedures necessary to follow the §91.175 regulation (i.e., for seeing the 
runway with "un-aided” vision by 100 ft HA T) were generally found to be awkward for the PF. 
The brightness on the HUD was selected by each PF for good readability, yet dim enough to 
promote viewability of the outside scene. However, the brightness of the Out-the- Window 
(OTW) visual scene was typically not sufficient to "burn-through” the imagery shown on the 
HUD, requiring the PF to declutter the HUD or to look-around the HUD combiner. (The 
brightness of the OTW w-as obviously not equivalent to real-wwld flight conditions, but "bum- 
through" of real-world imagery, while more likely in actual flight conditions, cannot be 
guaranteed.) The flight crew understood the rationale for the natural vision acquisition of the 
landing references, but the fact that the pilot flying had excellent visibility of the runw'ay using 
the FLIP made it very tempting to continue the approach to landing despite the rule. 

Adherence to the minimums was also hindered by crew duties. The radio altitude shown 
on the HUD could be used forjudging HAT, but the PF w'as concentrating primarily on lateral 
line-up and flare, focusing on the flight path marker and the image of the runway in the FL1R - 
not the displayed radar altimeter readout. The PNF was typically "eyes-out" and not closely 
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monitoring the altitude (shown on the head-down displays) below 200 ft AFL (Bailey, Kramer, 

& Prinzel, 2007). An aural call-out of 100 ft may have aided adherence to the DH. The aural 
call-outs were set to Cat. 1 decision heights (i.e., the aural call-out of "Minimums” occurs at 200 
ft AFL.). 

The few occurrences of "below minimums" landings suggest that the current regulations 
can be operationally viable. However, to further aid crew decision-making, an aural call-out at 
100 ft AFL is recommended. Nonetheless, the PF’s required transition from EV/HUD-to-visual 
runway references was awkward. The PFs typically commented that the EFVS provided suitable 
visual references to complete the flare and landing. 

Non-Normals 

Non-normals were injected into the test without the evaluation subjects having prior 
knowledge of them. The non-normals were two runway incursions and four lateral offset 
navigation errors for each flight crew. 

Runway Incursions 

The runway incursions were created by a baggage cart and a fire truck. While many 
issues factor into the data analysis for runway incursion, four of the key attributes that must be 
considered in analyzing these data are: display resolution, target size, display color (presence and 
absence), and the presence or absence of symbology. These factors will be discussed in relation 
to the experimental data. 

To put the R1 results into perspective of display resolution, the EVS and OTW 
(simulated) visual scene resolutions were used to compute the altitude AFL that the Rls might be 
reasonably expected to be observed by the crew. While target detection is critically dependent 
upon visibility, lighting, target contrast, color, gray scales, etc., resolution is only used in this 



Crew Decision-Making using Fused S/EVS 22 


example. In this example, it is assumed that 10 pixels (scan lines) are required for a human 
observer to recognize a target/object in this example (Larimer, Pavel, Ahumada. & Sweet, 1992). 
The baggage cart consisted of a tug and a cart. The tug was approximately 7.5 ft (2.3 in) tall and 
10 ft (3 m) long, tied to a cart 6.5 ft (2 m) tall and 10 ft (3 m) long. The Fire Truck was 3 1 .6 ft 
(9.6 in) long and 13 ft (4 in) tall. An operating rotating beacon was depicted atop the Fire Truck. 

In Table 3, the height of the aircraft (AFL) when 10 lines (pixels) draw the incurring 
vehicles is shown. The analysis assumes a 3 degree glideslope and the limiting resolution of the 
EVS shown on the HUD and PNF-AD was the simulated FLIP (640x480 resolution). The HUD 
and PNF-AD EVS resolution was 20 pixels per degree. The OTW resolution provides 26 pixels 
per degree. The vertical and horizontal resolutions were identical. 

Table 3 


Above Field Level (AFL) Altitudes for Theoretical Detection of Runway Incursions 


Object 

Dimension / 
Scan Direction 

AFL Altitudes for EVS 
“Detection" 

AFL Altitude for OTW 
“Detection” 

Baggage Cart 

Height 

42 It 

58 ft 


Width 

120ft 

156ft 

Fire Truck 

Height 

78 ft 

101 ft 


Width 

190 ft 

246 ft 


The theoretical detection ranges show that the Fire Truck should be detectable at almost 
twice the distance as the Baggage Cart, particularly its vertical extent (i.c., height). The OTW 
provided better detection capability because of its higher resolution (a 30% improvement). 
Finally, none of the vehicles were theoretically "detectable” in this analysis above 200 ft AFL 
using the EVS (i.c., using the HUD or PNF-AD). It should be noted, however, that the Fire 
Truck was visually detectable above 200 ft AFL using the EVS, // the observer was cued to its 
existence and studied the display. 
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The experimental results showed that, for the 12 flight crews, only one crew member 
(PNF) saw the baggage cart (they saw it in the OTW) and initiated a go-around. The other 1 1 
crews had a runway incursion with the baggage cart. From the analysis of Table 3, the baggage 
cart should be “detectable” between 50 and 150 ft AFL but this leaves only 5 to 15 seconds 
before landing - not very much time to spot a small object, parked slightly short of the intended 
touchdown zone. 

Eleven crews saw the Fire Truck OTW (7 by the PNF, 3 by the PF, and 1 simultaneously 
by the PF and PNF) and the one remaining crew saw it on the PNF-AD. Upon seeing the 
incursions, all crews initiated a go-around (all lower than 50 fl AFL). 

The incurring vehicles were visible in the PNF-AD and FIL'D, yet the data suggests that 
EV on the HUD and PNF-AD were not useful for R1 detection. Only one of the runway 
incursion scenarios was detected through use of the EVS imagery on the cockpit displays. These 
data highlight several important factors in R1 detection by flight crews: 

• the vehicles were stationary so motion effects which might aid target detection 
were absent. 

• the incurring vehicle scenarios were low luminance and low color contrast targets. 
Acuity metrics, such as those used in the example in Table 3, will provide 
optimistic detection performance since the metrics typically assume relatively 
high contrast targets. Whatever color contrast was available between the 
incurring vehicles and the runway background was only available to the pilots 
when using the OTW information for detection - not in the monochromatic EVS. 

• the flight path marker and guidance cue symbology on HUD or on the PNF-AD 
(when used) largely occluded the incurring vehicles as they were positioned near 
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the touchdown aimpoint. The ilight path marker shows precisely where the 
aircraft is going, but it will also obscure raster background imagery, thus, hiding a 
potential runway collision. 

• the experimental HUD, like all HUDs currently manufactured, has limited 

contrast (gray scale) ranges. Object detection is much more likely when using the 
PNF-AD than the HUD because of its significantly higher gray scale range. 
Further, the HUD image contrast was also a function of pilot control. 

Pilot duties are another important consideration on R! detection and in interpreting these 
experimental data. The PF was consumed with the task of Hying the aircraft and identifying the 
proper visual references for landing. Therefore, successful runway incursion detection at this 
late stage on the approach by the PF should not be considered as a likely outcome. The data 
confirmed this hypothesis in that the PF detected only 4 out of the 24 runway incursions. 

On the other hand, the PNF was tasked with monitoring the flight. Using the PNF-AD. 
the vehicles were much more apparent than in the HUD, but the data does not indicate 
significantly greater detection success using this display. The PNF-AD display is more attuned 
to the Rl detection task because of the lack of symbology (in some cases) and its improved gray 
scale contrast performance. But, as the theoretical detection example showed (Table 3), the 
vehicle size and EVS resolution make detection on the PNF-AD moderately difficult above 200 
ft AFL. Below 200 ft AFL, the vehicles were much more obvious in the image, but the PNFs 
noted that they were head-out the vast majority of the time. PNF head tracking measurements 
collected in this experiment (Bailey, et al., 2007) quantify this pilot comment. The PNF was 
head-out the vast majority of the time, ranging from 86% to 100% of the total time below 200 ft. 
AFL. Based on these data and the pilot comments, the use of the PNF-AD for incursion 
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detection was not probable. The presence of symbology on the PNF-AD could also have 
obscured the vehicles, particularly if the PNF only used cursory looks at the PNF-AD. It appears 
that the use of PNF Fusion controls when available (i.e., toggling between SV and EV imagery) 
did not statistically help or hinder detection (Bailey, et al„ 2007). 

Unlike the PF, the PNF is tasked with monitoring the approach and the landing area. 9 
out of 24 incursions were detected by the PNF. But, starting around 500 ft AFL and below, the 
PNF is going "eyes-out” so runway incursion detection using a head-down display may not be 
practical (only one PNF saw the incursion in the AD). The off-boresight design of the tested 
PNF-AD was noted by the pilots as hampering their use of the display on final. Current flight 
crews are not familiar with using head-down displays on short final to check for incursions. This 
was not part of the pre-experiment flight crew instructional briefing. 

The results suggest that EV on the HUD and PNF-AD were not useful for Rl detection. 
Requirements for display and sensor technology for runway incursion detection should be 
developed and they should span the breadth of the problem, including human perception, sensor 
design and detection theory, crew procedures, and crew interface issues. Current flight crews are 
not familiar with using head-down displays on short final to check for incursions. The displays 
are not necessarily optimized for this role. For instance, the capability to provide EV image 
“zoom" - to increase the EV resolution for object detection before the final phase of the approach 
- should be considered as it eould increase the probability for object detection and recognition 
before the PNF goes "eyes-out." 

Navigation Error 

The navigation errors were either a 50 A or 75 ft ( 15 m or 23 m) lateral offset (see Figure 
4) and eould be detected by either the PF or the PNF. The errors were noticeable from one of 
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several principal ways (depending upon the display configuration) using various indicators: 1 ) 
by a disagreement between the lateral path error and the localizer deviation symbology (HUD 
and PNF-AD concepts with symbology); 2) by a non-zero localizer deviation on the primary 
llight display (PFD) when the PF is flying on the final approach path centerline; 3) by 
differences between the SV and the EV registration using the PNF-AD Fusion controls; 4) by 
differences between the runway outline and the EV imagery of the runway (HUD and PNF-AD 
concepts with symbology); or 5) by differences in the pitch'roll guidance symbol and the EV 
imagery (PF-HUD and PNF-AD concepts with symbology). 


a) 50 ft L ateral offs et as seer on PF-HUD 


1: 1 75 f t Lateral offset as se e n on PNF-AD 



Figure 4. Lateral navigation offset examples. 

The majority of flight crews verbally noted the presence of the 50 ft ( 15 m) offset ( 1 5 of 
24 crews) and 75 ft (23 m) offset (19 of 24 crews) during the approach. None of the pilots 
executed a go-around with this anomaly. Each performed a lateral correction and landed near the 
runway centerline. 

Video analysis showed that navigation errors were predominately noted by the PF 
(-85%) when they noticed that the pitch’roll guidance symbol was leading them to the left or 
right of the runway. The "real" runway was detectable in the EV image. One person (flying as 
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the FNF) noted the non-zero localizer deviation on the PFD presentation while tracking the path 
centerline. 

The flight crews were not instructed on the course of action to take when confronted with 
a navigation error, and the pilots had relatively little training and experience with the system. 
Despite this, the study showed that lateral navigation errors were verbally acknowledged a 
significant percentage of time, and even w hen unrecognized (i.e., not explicitly verbalized), all 
flight crews landed safely and accurately on the runway. These results suggest that dissociations 
between raw' data, sensor, and/or database presentation should be easily recognized and managed 
by experienced pilots. Pilot training to recognize these discrepancies should further improve 
operations in the event of this anomaly. 

Conclusions 

An experiment w ( as conducted to evaluate the complementary use of synthetic vision and 
enhanced vision technologies, specifically evaluating the utility, acceptability, and usability of 
integrated/fused enhanced and synthetic vision technologies and its effect on two-crew decision- 
making during low-visibility approach and landing operations. Pilots were asked to fly RNP- 
type approaches under conditions of restricted visibility to evaluate the efficacy of synthetic and 
enhanced vision concepts for future equivalent visual NcxtGen operations. The research further 
evaluated the effect of these technologies on flight crew' responses to variable weather conditions 
and selected rare events that may impact aviation safety during such operations. Kramer, Bailey, 
and Prinzel (2007) present nominal results and discuss the potential of these technologies for 
supporting future national airspace system operations. The current paper describes the off- 
nominal results on flight crew responses to below minimum landing conditions (i.e., “illegal 
landings”) and pilot reactions to runway incursions and navigational offset error "rare events." 
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Illegal Landings 

The few occurrences of "below minimuras” landings suggest that the current Federal 
Aviation Regulation (FAR) $91,175 rules can be operationally viable with these displays. 
However, to further aid crew decision-making, an aural call-out at 100 ft above field level is 
recommended. The $91,175 regulation requires that the pilot-flying (PF) see the required vision 
landing references with natural vision by 100 ft Height Above Touchdown. This natural vision 
(versus enhanced vision) requirement caused an awkwardness for the PF in the late stages of the 
approach because it required the PF to declutter the head-up display (HUD) or to look-around the 
HUD combiner. The procedural requirement added a workload cost to the use of the FVS and 
added to the temporal demands imposed on the flight crew during final approach. Pilot 
commentary indicated that the Enhanced Flight Visual System (EFVS) - forward looking 
infrared (FL1R) on a HUD - provided suitable visual references to complete the flare and 
landing. Future research should examine the effects of potentially eliminating the natural 
vision'visual segment requirement of FAR $91,175 in other than Category 11 or 111 operations 
while conducting low visibility approach and landing with an EFVS. 

Rare Events 

The ability of the flight crew to handle a runway incursion was neither impacted nor 
significantly aided by the display concepts tested. Although the increase in near-domain 
symbology information (runway outline) did not degrade pilot response to the Fire Truck runway 
incursion event, there was also not an observed enhancement in incursion detection as 
hypothesized for the FLIR. The display concepts and scenarios tested in this experiment - 
typical of current and future PF HUD and pilot-not- flying auxiliary displays - did not show 
adequate incursion detection functionality. Numerous factors contributed to this result including 
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the location and size of the incursion vehicle which somewhat blended into the runway markings 
and was occluded by flight display symbology. As a consequence, only one of the runway 
incursion scenarios was detected using the cockpit displays, which included EV. These results 
do not necessarily suggest that the EV has no value in runway incursion detection. However, the 
imaging sensor alone does not provide sufficient capability to significantly and reliably prevent 
occurrence of these events. Therefore, sensor and display design must be tailored to this 
function and corresponding crew procedures and interfaces developed to support runway 
incursion detection. 

In addition to runway incursion detection, another rare event scenario evaluated the 
ability of the flight crew to perform their required crew procedures and handle a substantial 
navigation solution error. The lateral navigation errors were verbally acknowledged a significant 
percentage of time, and had minimal, if any, impact on the flight crews ability to make the 
necessary adjustments to land safely and accurately on the runway. These results lend 
convincing evidence that the potentially compelling display of pathway information (in the form 
of a runway outline in this test) does not adversely capture attention nor induce pilots to follow 
erroneous display information in the presence of real-world visual information or other display 
cross-checks. 

The emerging NcxtGcn operational concepts will require revolutionary approaches to 
help meet the many new challenges of the future air transportation system. Synthetic and 
enhanced vision systems offer promising enabling technologies to proactively overcome aircraft 
safety barriers that would otherwise constrain the full realization of the next generation air 
transportation system. This research should lead toward the development of new operating 
concepts - such as 4D navigation and self-separation (see Joint Planning and Development 
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Office, 2004) - as well as all-wealhcr operations capabilities, approaching that which might 
create an equivalent visual operations capability, yet without necessarily requiring the magnitude 
of airport or aircraft infrastructure associated with today's instrument flight operations. 
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